Follow-Me Project

Congratulations on reaching the final project of the Robotics Nanodegree!

Previously, you worked on the Semantic Segmentation lab where you built a deep learning network that locates a particular human target within an image. For this project, you will utilize what you implemented and learned from that lab and extend it to train a deep learning model that will allow a simulated quadcopter to follow around the person that it detects!

Most of the code below is similar to the lab with some minor modifications. You can start with your existing solution, and modify and improve upon it to train the best possible model for this task.

You can click on any of the following to quickly jump to that part of this notebook:

  1. Data Collection
  2. FCN Layers
  3. Build the Model
  4. Training
  5. Prediction
  6. Evaluation

Data Collection

We have provided you with a starting dataset for this project. Download instructions can be found in the README for this project's repo. Alternatively, you can collect additional data of your own to improve your model. Check out the "Collecting Data" section in the Project Lesson in the Classroom for more details!

In [4]:
import os
import glob
import sys
import tensorflow as tf

from scipy import misc
import numpy as np

from tensorflow.contrib.keras.python import keras
from tensorflow.contrib.keras.python.keras import layers, models
from tensorflow.contrib.keras.python.keras.callbacks import ModelCheckpoint

from tensorflow import image

from utils import scoring_utils
from utils.separable_conv2d import SeparableConv2DKeras, BilinearUpSampling2D
from utils import data_iterator
from utils import plotting_tools 
from utils import model_tools

import pandas as pd
# from keras.callbacks import CSVLogger, EarlyStopping, ModelCheckpoint, ReduceLROnPlateau

from keras_tqdm import TQDMNotebookCallback

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

%matplotlib inline
%config InlineBackend.figure_format='retina'
%config IPCompleter.greedy=True

FCN Layers

In the Classroom, we discussed the different layers that constitute a fully convolutional network (FCN). The following code will introduce you to the functions that you need to build your semantic segmentation model.

Separable Convolutions

The Encoder for your FCN will essentially require separable convolution layers, due to their advantages as explained in the classroom. The 1x1 convolution layer in the FCN, however, is a regular convolution. Implementations for both are provided below for your use. Each includes batch normalization with the ReLU activation function applied to the layers.

In [5]:
def separable_conv2d_batchnorm(input_layer, filters, strides=1):
    output_layer = SeparableConv2DKeras(filters=filters,kernel_size=3, strides=strides,
                             padding='same', activation='relu')(input_layer)
    
    output_layer = layers.BatchNormalization()(output_layer) 
    return output_layer

def conv2d_batchnorm(input_layer, filters, kernel_size=3, strides=1):
    output_layer = layers.Conv2D(filters=filters, kernel_size=kernel_size, strides=strides, 
                      padding='same', activation='relu')(input_layer)
    
    output_layer = layers.BatchNormalization()(output_layer) 
    return output_layer

Bilinear Upsampling

The following helper function implements the bilinear upsampling layer. Upsampling by a factor of 2 is generally recommended, but you can try out different factors as well. Upsampling is used in the decoder block of the FCN.

In [6]:
def bilinear_upsample(input_layer):
    output_layer = BilinearUpSampling2D((2,2))(input_layer)
    return output_layer

Build the Model

In the following cells, you will build an FCN to train a model to detect and locate the hero target within an image. The steps are:

  • Create an encoder_block
  • Create a decoder_block
  • Build the FCN consisting of encoder block(s), a 1x1 convolution, and decoder block(s). This step requires experimentation with different numbers of layers and filter sizes to build your model.

Encoder Block

Create an encoder block that includes a separable convolution layer using the separable_conv2d_batchnorm() function. The filters parameter defines the size or depth of the output layer. For example, 32 or 64.

In [7]:
def encoder_block(input_layer, filters, strides):
    
#     print("encoder_block::input_layer::",input_layer.shape)
    
    # TODO Create a separable convolution layer using the separable_conv2d_batchnorm() function.
    input_layer = SeparableConv2DKeras(filters=filters,kernel_size=3, strides=1,
                             padding='same', activation='relu')(input_layer)
    
#     print("encoder_block::input_layer2::",input_layer.shape)
    
    output_layer = separable_conv2d_batchnorm(input_layer=input_layer, filters=filters, strides=strides)
    
#     print("encoder_block::output_layer::",output_layer.shape)

    
    return output_layer

Decoder Block

The decoder block is comprised of three parts:

  • A bilinear upsampling layer using the upsample_bilinear() function. The current recommended factor for upsampling is set to 2.
  • A layer concatenation step. This step is similar to skip connections. You will concatenate the upsampled small_ip_layer and the large_ip_layer.
  • Some (one or two) additional separable convolution layers to extract some more spatial information from prior layers.
In [8]:
def decoder_block(small_ip_layer, large_ip_layer, filters):
    
    # TODO Upsample the small input layer using the bilinear_upsample() function.
    upsample_layer = bilinear_upsample(small_ip_layer)
    
    # TODO Concatenate the upsampled and large input layers using layers.concatenate
    concat_layer = layers.concatenate(inputs=[upsample_layer, large_ip_layer])
    
    # TODO Add some number of separable convolution layers
    output_layer = separable_conv2d_batchnorm(input_layer=concat_layer, filters=filters)
        
    return output_layer

Model

Now that you have the encoder and decoder blocks ready, go ahead and build your FCN architecture!

There are three steps:

  • Add encoder blocks to build the encoder layers. This is similar to how you added regular convolutional layers in your CNN lab.
  • Add a 1x1 Convolution layer using the conv2d_batchnorm() function. Remember that 1x1 Convolutions require a kernel and stride of 1.
  • Add decoder blocks for the decoder layers.
In [9]:
def fcn_model(inputs, num_classes):
    
    # TODO Add Encoder Blocks. 
    # Remember that with each encoder layer, the depth of your model (the number of filters) increases.
    print("Inputs : ", inputs.shape)

#     l0 = SeparableConv2DKeras(filters=8,kernel_size=3, strides=1,
#                              padding='same', activation='relu')(inputs)
    
#     print("L0 : ", l0.shape)
    l1 = encoder_block(inputs, filters=8, strides=2)
    print("L1 : ", l1.shape)
    l2 = encoder_block(l1, filters=16, strides=2)
    print("L2 : ", l2.shape)
    l3 = encoder_block(l2, filters=32, strides=2)
    print("L3 : ", l3.shape)
    l3_2 = encoder_block(l3, filters=64, strides=2)
    print("L3_2 : ", l3_2.shape)
    l3_3 = encoder_block(l3_2, filters=128, strides=2)
    print("L3_3 : ", l3_3.shape)

    # TODO Add 1x1 Convolution layer using conv2d_batchnorm().
    l4 = conv2d_batchnorm(l3_3, filters=256, kernel_size=1)
    print("L4 : ", l4.shape)
    
    # TODO: Add the same number of Decoder Blocks as the number of Encoder Blocks
    l5_3 = decoder_block(l4, l3_2, filters=128)
    print("L5_3 : ", l5_3.shape)
    l5_2 = decoder_block(l5_3, l3, filters=64)
    print("L5_2 : ", l5_2.shape)
    l5 = decoder_block(l5_2, l2, filters=32)
    print("L5 : ", l5.shape)
    l6 = decoder_block(l5, l1, filters=16)
    print("L6 : ", l6.shape)
    l7 = decoder_block(l6, inputs, filters=8)
    print("L7 : ", l7.shape)
    
    # The function returns the output layer of your model. "x" is the final layer obtained from the last decoder_block()
    output_layer = layers.Conv2D(num_classes, 3, activation='softmax', padding='same')(l7)
    print("Output : ", output_layer.shape)
    
    
    return output_layer

Training

The following cells will use the FCN you created and define an ouput layer based on the size of the processed image and the number of classes recognized. You will define the hyperparameters to compile and train your model.

Please Note: For this project, the helper code in data_iterator.py will resize the copter images to 160x160x3 to speed up training.

In [10]:
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

image_hw = 256
image_shape = (image_hw, image_hw, 3)
inputs = layers.Input(image_shape)
num_classes = 3

# Call fcn_model()
output_layer = fcn_model(inputs, num_classes)
Out[10]:
"\nDON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n"
Inputs :  (?, 256, 256, 3)
L1 :  (?, 128, 128, 8)
L2 :  (?, 64, 64, 16)
L3 :  (?, 32, 32, 32)
L3_2 :  (?, 16, 16, 64)
L3_3 :  (?, 8, 8, 128)
L4 :  (?, 8, 8, 256)
L5_3 :  (?, 16, 16, 128)
L5_2 :  (?, 32, 32, 64)
L5 :  (?, 64, 64, 32)
L6 :  (?, 128, 128, 16)
L7 :  (?, 256, 256, 8)
Output :  (?, 256, 256, 3)

Hyperparameters

Define and tune your hyperparameters.

  • batch_size: number of training samples/images that get propagated through the network in a single pass.
  • num_epochs: number of times the entire training dataset gets propagated through the network.
  • steps_per_epoch: number of batches of training images that go through the network in 1 epoch. We have provided you with a default value. One recommended value to try would be based on the total number of images in training dataset divided by the batch_size.
  • validation_steps: number of batches of validation images that go through the network in 1 epoch. This is similar to steps_per_epoch, except validation_steps is for the validation dataset. We have provided you with a default value for this as well.
  • workers: maximum number of processes to spin up. This can affect your training speed and is dependent on your hardware. We have provided a recommended value to work with.
In [42]:
learning_rate = 0.001
batch_size = 32
num_epochs = 50
steps_per_epoch = 250
validation_steps = 100
workers = 2
In [12]:
# Define the Keras model and compile it for training
model = models.Model(inputs=inputs, outputs=output_layer)
In [13]:
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         (None, 256, 256, 3)       0         
_________________________________________________________________
separable_conv2d_keras_1 (Se (None, 256, 256, 8)       59        
_________________________________________________________________
separable_conv2d_keras_2 (Se (None, 128, 128, 8)       144       
_________________________________________________________________
batch_normalization_1 (Batch (None, 128, 128, 8)       32        
_________________________________________________________________
separable_conv2d_keras_3 (Se (None, 128, 128, 16)      216       
_________________________________________________________________
separable_conv2d_keras_4 (Se (None, 64, 64, 16)        416       
_________________________________________________________________
batch_normalization_2 (Batch (None, 64, 64, 16)        64        
_________________________________________________________________
separable_conv2d_keras_5 (Se (None, 64, 64, 32)        688       
_________________________________________________________________
separable_conv2d_keras_6 (Se (None, 32, 32, 32)        1344      
_________________________________________________________________
batch_normalization_3 (Batch (None, 32, 32, 32)        128       
_________________________________________________________________
separable_conv2d_keras_7 (Se (None, 32, 32, 64)        2400      
_________________________________________________________________
separable_conv2d_keras_8 (Se (None, 16, 16, 64)        4736      
_________________________________________________________________
batch_normalization_4 (Batch (None, 16, 16, 64)        256       
_________________________________________________________________
separable_conv2d_keras_9 (Se (None, 16, 16, 128)       8896      
_________________________________________________________________
separable_conv2d_keras_10 (S (None, 8, 8, 128)         17664     
_________________________________________________________________
batch_normalization_5 (Batch (None, 8, 8, 128)         512       
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 8, 8, 256)         33024     
_________________________________________________________________
batch_normalization_6 (Batch (None, 8, 8, 256)         1024      
_________________________________________________________________
bilinear_up_sampling2d_1 (Bi (None, 16, 16, 256)       0         
_________________________________________________________________
concatenate_1 (Concatenate)  (None, 16, 16, 320)       0         
_________________________________________________________________
separable_conv2d_keras_11 (S (None, 16, 16, 128)       43968     
_________________________________________________________________
batch_normalization_7 (Batch (None, 16, 16, 128)       512       
_________________________________________________________________
bilinear_up_sampling2d_2 (Bi (None, 32, 32, 128)       0         
_________________________________________________________________
concatenate_2 (Concatenate)  (None, 32, 32, 160)       0         
_________________________________________________________________
separable_conv2d_keras_12 (S (None, 32, 32, 64)        11744     
_________________________________________________________________
batch_normalization_8 (Batch (None, 32, 32, 64)        256       
_________________________________________________________________
bilinear_up_sampling2d_3 (Bi (None, 64, 64, 64)        0         
_________________________________________________________________
concatenate_3 (Concatenate)  (None, 64, 64, 80)        0         
_________________________________________________________________
separable_conv2d_keras_13 (S (None, 64, 64, 32)        3312      
_________________________________________________________________
batch_normalization_9 (Batch (None, 64, 64, 32)        128       
_________________________________________________________________
bilinear_up_sampling2d_4 (Bi (None, 128, 128, 32)      0         
_________________________________________________________________
concatenate_4 (Concatenate)  (None, 128, 128, 40)      0         
_________________________________________________________________
separable_conv2d_keras_14 (S (None, 128, 128, 16)      1016      
_________________________________________________________________
batch_normalization_10 (Batc (None, 128, 128, 16)      64        
_________________________________________________________________
bilinear_up_sampling2d_5 (Bi (None, 256, 256, 16)      0         
_________________________________________________________________
concatenate_5 (Concatenate)  (None, 256, 256, 19)      0         
_________________________________________________________________
separable_conv2d_keras_15 (S (None, 256, 256, 8)       331       
_________________________________________________________________
batch_normalization_11 (Batc (None, 256, 256, 8)       32        
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 256, 256, 3)       219       
=================================================================
Total params: 133,185
Trainable params: 131,681
Non-trainable params: 1,504
_________________________________________________________________

Using the newly distributed dataset with horizontl flip augmentation only...

In [43]:
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
# Define the Keras model and compile it for training
# model = models.Model(inputs=inputs, outputs=output_layer)

model.compile(optimizer=keras.optimizers.Adam(learning_rate), loss='categorical_crossentropy')

# Data iterators for loading the training and validation data
train_iter = data_iterator.BatchIteratorSimple(batch_size=batch_size,
                                               data_folder=os.path.join('..', 'data', 'train'),
                                               image_shape=image_shape,
                                               shift_aug=True)

val_iter = data_iterator.BatchIteratorSimple(batch_size=batch_size,
                                             data_folder=os.path.join('..', 'data', 'validation'),
                                             image_shape=image_shape)

logger_cb = plotting_tools.LoggerPlotter()
callbacks = [logger_cb]
callbacks.append(ModelCheckpoint(filepath='../data/weights/5_layers_29/1/unet_{epoch:02d}_{loss:.3f}_{val_loss:.3f}',
                             monitor = 'val_loss', verbose=1, save_best_only=False, save_weights_only=True))
# callbacks.append(CSVLogger('../data/logs/unet_128_history.csv'))
# callbacks.append(EarlyStopping(monitor='val_loss', patience=8, verbose=1, min_delta=1e-4))
# callbacks.append(ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=4, verbose=1, epsilon=1e-4))
callbacks.append(TQDMNotebookCallback())

history = model.fit_generator(train_iter,
                    steps_per_epoch = steps_per_epoch, # the number of batches per epoch,
                    epochs = num_epochs, # the number of epochs to train for,
                    validation_data = val_iter, # validation iterator
                    validation_steps = validation_steps, # the number of batches to validate on
                    callbacks=callbacks,
                    workers = workers, verbose = 0)
Out[43]:
"\nDON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n"
Epoch 00000: saving model to ../data/weights/5_layers_29/1/unet_00_0.403_0.100
Epoch 00001: saving model to ../data/weights/5_layers_29/1/unet_01_0.049_0.093
Epoch 00002: saving model to ../data/weights/5_layers_29/1/unet_02_0.039_0.047
Epoch 00003: saving model to ../data/weights/5_layers_29/1/unet_03_0.033_0.045
Epoch 00004: saving model to ../data/weights/5_layers_29/1/unet_04_0.034_0.049
Epoch 00005: saving model to ../data/weights/5_layers_29/1/unet_05_0.031_0.037
Epoch 00006: saving model to ../data/weights/5_layers_29/1/unet_06_0.027_0.033
Epoch 00007: saving model to ../data/weights/5_layers_29/1/unet_07_0.026_0.034
Epoch 00008: saving model to ../data/weights/5_layers_29/1/unet_08_0.024_0.031
Epoch 00009: saving model to ../data/weights/5_layers_29/1/unet_09_0.022_0.036
Epoch 00010: saving model to ../data/weights/5_layers_29/1/unet_10_0.022_0.031
Epoch 00011: saving model to ../data/weights/5_layers_29/1/unet_11_0.021_0.035
Epoch 00012: saving model to ../data/weights/5_layers_29/1/unet_12_0.020_0.031
Epoch 00013: saving model to ../data/weights/5_layers_29/1/unet_13_0.019_0.038
Epoch 00014: saving model to ../data/weights/5_layers_29/1/unet_14_0.019_0.029
Epoch 00015: saving model to ../data/weights/5_layers_29/1/unet_15_0.018_0.029
Epoch 00016: saving model to ../data/weights/5_layers_29/1/unet_16_0.017_0.026
Epoch 00017: saving model to ../data/weights/5_layers_29/1/unet_17_0.016_0.032
Epoch 00018: saving model to ../data/weights/5_layers_29/1/unet_18_0.017_0.030
Epoch 00019: saving model to ../data/weights/5_layers_29/1/unet_19_0.015_0.027
Epoch 00020: saving model to ../data/weights/5_layers_29/1/unet_20_0.015_0.034
Epoch 00021: saving model to ../data/weights/5_layers_29/1/unet_21_0.014_0.028
Epoch 00022: saving model to ../data/weights/5_layers_29/1/unet_22_0.016_0.026
Epoch 00023: saving model to ../data/weights/5_layers_29/1/unet_23_0.014_0.044
Epoch 00024: saving model to ../data/weights/5_layers_29/1/unet_24_0.014_0.029
Epoch 00025: saving model to ../data/weights/5_layers_29/1/unet_25_0.015_0.032
Epoch 00026: saving model to ../data/weights/5_layers_29/1/unet_26_0.015_0.026
Epoch 00027: saving model to ../data/weights/5_layers_29/1/unet_27_0.014_0.032
Epoch 00028: saving model to ../data/weights/5_layers_29/1/unet_28_0.013_0.028
Epoch 00029: saving model to ../data/weights/5_layers_29/1/unet_29_0.012_0.028
Epoch 00030: saving model to ../data/weights/5_layers_29/1/unet_30_0.012_0.053
Epoch 00031: saving model to ../data/weights/5_layers_29/1/unet_31_0.012_0.025
Epoch 00032: saving model to ../data/weights/5_layers_29/1/unet_32_0.011_0.035
Epoch 00033: saving model to ../data/weights/5_layers_29/1/unet_33_0.011_0.031
Epoch 00034: saving model to ../data/weights/5_layers_29/1/unet_34_0.011_0.031
Epoch 00035: saving model to ../data/weights/5_layers_29/1/unet_35_0.011_0.025
Epoch 00036: saving model to ../data/weights/5_layers_29/1/unet_36_0.010_0.027
Epoch 00037: saving model to ../data/weights/5_layers_29/1/unet_37_0.010_0.034
Epoch 00038: saving model to ../data/weights/5_layers_29/1/unet_38_0.011_0.044
Epoch 00039: saving model to ../data/weights/5_layers_29/1/unet_39_0.015_0.036
Epoch 00040: saving model to ../data/weights/5_layers_29/1/unet_40_0.012_0.028
Epoch 00041: saving model to ../data/weights/5_layers_29/1/unet_41_0.011_0.027
Epoch 00042: saving model to ../data/weights/5_layers_29/1/unet_42_0.010_0.026
Epoch 00043: saving model to ../data/weights/5_layers_29/1/unet_43_0.010_0.026
Epoch 00044: saving model to ../data/weights/5_layers_29/1/unet_44_0.010_0.032
Epoch 00045: saving model to ../data/weights/5_layers_29/1/unet_45_0.010_0.027
Epoch 00046: saving model to ../data/weights/5_layers_29/1/unet_46_0.011_0.029
Epoch 00047: saving model to ../data/weights/5_layers_29/1/unet_47_0.013_0.037
Epoch 00048: saving model to ../data/weights/5_layers_29/1/unet_48_0.011_0.029
Epoch 00049: saving model to ../data/weights/5_layers_29/1/unet_49_0.009_0.024
In [ ]:
callbacks = [logger_cb]
callbacks.append(ModelCheckpoint(filepath='../data/weights/5_layers_27/2/unet_256.{epoch:02d}-{val_loss:.3f}',
                             monitor = 'val_loss', verbose=1, save_best_only=False, save_weights_only=True))
# callbacks.append(CSVLogger('../data/logs/unet_128_history.csv'))
# callbacks.append(EarlyStopping(monitor='val_loss', patience=8, verbose=1, min_delta=1e-4))
# callbacks.append(ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=4, verbose=1, epsilon=1e-4))
callbacks.append(TQDMNotebookCallback())

history2 = model.fit_generator(train_iter,
                    steps_per_epoch = steps_per_epoch, # the number of batches per epoch,
                    epochs = num_epochs, # the number of epochs to train for,
                    validation_data = val_iter, # validation iterator
                    validation_steps = validation_steps, # the number of batches to validate on
                    callbacks=callbacks,
                    workers = workers, verbose = 0)
In [44]:
# import pandas as pd
# Save your trained model weights
weight_file_name = 'model_weights_5layer_29_1'
model_tools.save_network(model, weight_file_name)
history_DF_3 = pd.DataFrame(history.history)
history_DF_3.to_csv('../data/logs/{}.csv'.format(weight_file_name))
In [ ]:
history_DF = pd.DataFrame(history.history)
history_DF.to_csv('../data/logs/{}.csv'.format('model_weights_new_5layer_6_1'))
In [ ]:
history_DF = pd.read_csv('../data/logs/{}.csv'.format(weight_file_name))
history_DF = history_DF.drop(history_DF.columns[0], 1)
history_DF.plot()
In [ ]:
# history_DF_2.plot()
history_DF_c = history_DF_c.append(history_DF_3, ignore_index = True)
history_DF_c.plot()

Prediction

Now that you have your model trained and saved, you can make predictions on your validation dataset. These predictions can be compared to the mask images, which are the ground truth labels, to evaluate how well your model is doing under different conditions.

There are three different predictions available from the helper code provided:

  • patrol_with_targ: Test how well the network can detect the hero from a distance.
  • patrol_non_targ: Test how often the network makes a mistake and identifies the wrong person as the target.
  • following_images: Test how well the network can identify the target while following them.
In [ ]:
# If you need to load a model which you previously trained you can uncomment the codeline that calls the function below.

weight_file_name = 'unet_256_10-0_02_6_b'
model = model_tools.load_network(weight_file_name)

The following cell will write predictions to files and return paths to the appropriate directories. The run_num parameter is used to define or group all the data for a particular model run. You can change it for different runs. For example, 'run_1', 'run_2' etc.

In [45]:
run_num = 'run_8_5layer_29_1'

val_with_targ, pred_with_targ = model_tools.write_predictions_grade_set(model,
                                        run_num,'patrol_with_targ', 'sample_evaluation_data') 

val_no_targ, pred_no_targ = model_tools.write_predictions_grade_set(model, 
                                        run_num,'patrol_non_targ', 'sample_evaluation_data') 

val_following, pred_following = model_tools.write_predictions_grade_set(model,
                                        run_num,'following_images', 'sample_evaluation_data')

Now lets look at your predictions, and compare them to the ground truth labels and original images. Run each of the following cells to visualize some sample images from the predictions in the validation set.

In [52]:
# images while following the target
im_files = plotting_tools.get_im_file_sample('sample_evaluation_data','following_images', run_num) 
for i in range(3):
    im_tuple = plotting_tools.load_images(im_files[i])
    plotting_tools.show_images(im_tuple)
    
In [53]:
# images while at patrol without target
im_files = plotting_tools.get_im_file_sample('sample_evaluation_data','patrol_non_targ', run_num) 
for i in range(3):
    im_tuple = plotting_tools.load_images(im_files[i])
    plotting_tools.show_images(im_tuple)
 
In [54]:
   
# images while at patrol with target
im_files = plotting_tools.get_im_file_sample('sample_evaluation_data','patrol_with_targ', run_num) 
for i in range(3):
    im_tuple = plotting_tools.load_images(im_files[i])
    plotting_tools.show_images(im_tuple)

Evaluation

Evaluate your model! The following cells include several different scores to help you evaluate your model under the different conditions discussed during the Prediction step.

In [46]:
# Scores for while the quad is following behind the target. 
true_pos1, false_pos1, false_neg1, iou1 = scoring_utils.score_run_iou(val_following, pred_following)
number of validation samples intersection over the union evaulated on 542
average intersection over union for background is 0.996451379351006
average intersection over union for other people is 0.41015285239406646
average intersection over union for the hero is 0.939277970402923
number true positives: 539, number false positives: 0, number false negatives: 0
In [47]:
# Scores for images while the quad is on patrol and the target is not visable
true_pos2, false_pos2, false_neg2, iou2 = scoring_utils.score_run_iou(val_no_targ, pred_no_targ)
number of validation samples intersection over the union evaulated on 270
average intersection over union for background is 0.9893179182148905
average intersection over union for other people is 0.7963411472070311
average intersection over union for the hero is 0.0
number true positives: 0, number false positives: 20, number false negatives: 0
In [48]:
# This score measures how well the neural network can detect the target from far away
true_pos3, false_pos3, false_neg3, iou3 = scoring_utils.score_run_iou(val_with_targ, pred_with_targ)
number of validation samples intersection over the union evaulated on 322
average intersection over union for background is 0.9970421587302861
average intersection over union for other people is 0.47537624481865437
average intersection over union for the hero is 0.1960762683711165
number true positives: 129, number false positives: 0, number false negatives: 183
In [49]:
# Sum all the true positives, etc from the three datasets to get a weight for the score
true_pos = true_pos1 + true_pos2 + true_pos3
false_pos = false_pos1 + false_pos2 + false_pos3
false_neg = false_neg1 + false_neg2 + false_neg3

weight = true_pos/(true_pos+false_neg+false_pos)
print(weight)
0.7669345579793341
In [50]:
# The IoU for the dataset that never includes the hero is excluded from grading
final_IoU = (iou1 + iou3)/2
print(final_IoU)
0.567677119387
In [51]:
# And the final grade score is 
final_score = final_IoU * weight
print(final_score)
0.435371200632
In [ ]: